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Abstract To maximize impact across the broad spectrum 
of mental health needs exhibited by youth in school set- 
tings, interventions must be designed to be effective and 
efficient and demonstrate good fit with the educational 
context. The current paper reports on the second phase of 
an iterative development process for a short-term “Tier 2” 
intervention for use by school-based mental health provi- 
ders—the Brief Intervention for School Clinicians 
(BRISC)—using mixed qualitative and quantitative analy- 
ses to evaluate feasibility, acceptability, and appropriate- 
ness while emphasizing student experiences. This phase 
was intended to yield information to drive further protocol 
refinement and testing across subsequent phases. We 
describe the rationale for, development of, and formative 
testing of the BRISC intervention. Results suggest that 
BRISC may be feasible to deliver, acceptable to students, 
and appropriate to the school context. In particular, the 
BRISC process appears to be effective in enhancing stu- 
dent engagement in the intervention and identifying and 
addressing individualized student needs. These findings 
and directions for further enhancing BRISC’s potential for 
positive impact highlight how treatment development may 
benefit from initial, small-scale evaluations focused on 
both client and implementation outcomes. 
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Introduction 


Schools offer significant advantages when conducting 
observation, screening, and assessment of children and 
adolescents’ emotions and _ behavior (McCormick, 
Thompson, Vander Stoep, & McCauley, 2009; Owens & 
Murphy, 2004), as well as the provision of accessible and 
timely mental health services (Kataoka, Stein, Nadeem, & 
Wong, 2007; Lyon, Ludwig, Vander Stoep, Gudmundsen, 
& McCauley, 2013). As a method of organizing and 
summarizing the range of available interventions that may 
benefit all students with academic challenges, schools are 
increasingly adopting tiered frameworks that present 
strategies at multiple levels, including universal (Tier 1, the 
entire school population), selective (Tier 2, students 
demonstrating emerging problems or heightened risk), and 
indicated levels (Tier 3, focused on students with needs 
that require intervention) (Fazel, Hoagwood, Stephan, & 
Ford, 2014). 

Results of meta-analyses show that universal mental 
health promotion programs can confer significant positive 
effects on both socio-emotional and academic outcomes of 
students (Payton et al., 2008). Whole-school- and class- 
room-based programs that promote better classroom 
behavior also have increasing empirical support (Brad- 
shaw, Koth, Bevans, Ialongo, & Leaf, 2008; Embry, 2002; 
Horner et al., 2009; Shure, 1992). Evidence is also growing 
for the effectiveness of selective prevention programs 
focused on aggressive behavior, substance abuse, anxiety, 
and depression (Fazel, Hoagwood, Stephan, & Ford 2014). 

School mental health (SMH) services are typically 
conceived as residing within Tiers 2 and 3, which comprise 
indicated school-based programs and interventions for 
students with identified need. Although there are some 
empirically supported, school-specific interventions for 
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certain diagnoses (e.g., post-traumatic stress disorder; Stein 
et al., 2003), few models for delivering SMH services to 
students with diverse needs have been rigorously exam- 
ined. Moreover, SMH scholars have consistently concluded 
that services for emotional and behavioral needs offered in 
the education sector are not sufficiently based on empirical 
evidence (Evans & Weist, 2004; Owens et al., 2014; Rones 
& Hoagwood, 2000). Even when high-quality evidence- 
based practices (EBP) are introduced into schools, they are 
most often characterized by “incomplete implementation, 
restricted sustainability, and narrow spread” (Fazel, 
Hoagwood, Stephan, & Ford 2014, p. 382). 

These implementation challenges are not unique to the 
school setting—implementation of EBP in other child- 
serving settings (e.g., community mental health) has also 
been limited (Becker, Smith, & Jensen-Doss, 2013; Hor- 
witz et al., 2014). However, there may be unique chal- 
lenges in school settings that further complicate the 
effective use of EBPs (Forman et al., 2013; Owens et al., 
2014). Barriers to EBP use in SMH include difficulties 
integrating mental health services with the core academic 
mission of education systems, obtaining support or buy in 
from various school personnel, training adequacy, com- 
peting priorities or lack of time for training and imple- 
mentation activities, and inadequate resources (Forman & 
Barakat, 2011; Forman, Olin, Hoagwood, Crowe, & Saka, 
2009; Lyon et al., 2013c). At the policy level, initiatives 
such as No Child Left Behind, Common Core, and “high- 
stakes” testing have increased schools’ emphasis on edu- 
cational outcomes and led to a corresponding decrease in 
emphasis on the socio-emotional outcomes targeted by 
most mental health EBPs. Further, given that few of the 
interventions used in SMH have been designed explicitly 
for the educational context (Wong, 2008), there are many 
unanswered questions surrounding the fit between mental 
health EBPs and the school setting and the degree to which 
SMH services actually promote academic outcomes. 

The current paper describes the rationale for, develop- 
ment of, and formative testing of a SMH intervention 
approach (the Brief Intervention for School Clinicians 
[BRISC]; Lyon et al., 2014a) aimed at addressing barriers 
to effective SMH service delivery. It is a school-based, 
stepped-care model designed to effectively assess and 
triage students while identifying an immediate and specific 
problem that students can actively address. It provides a 
flexible structure with up to four sessions to assess, engage, 
identify, and begin to address student challenges that are 
distressing and impacting academic performance/behavior 
and overall functioning. The goals of the paper are twofold: 
(a) to report results from an initial evaluation focused on 
both the potential for positive effects and implementation 
factors that are critical in crossing the “clinic to school 
chasm,” such as feasibility, acceptability, and contextual 
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appropriateness; (b) to provide an example of how treat- 
ment development may benefit from initial, small-scale 
evaluations focused on both client and implementation 
outcomes. 


Toward Better Student Outcomes: Brief 
Intervention for School Clinicians (BRISC) 


In light of the above impediments to the effective use of 
evidence in SMH, increasing work is being conducted to 
both (a) design effective interventions with good fit to the 
school context and (b) facilitate high-quality implementation 
of effective and promising practices (Forman et al., 2013; 
Owens et al., 2014). Regarding fit to context, SMH service 
delivery is characterized by large caseloads, significant 
clinician time constraints, and frequent student crises (Lyon 
et al., 2014b; Lyon et al., 2013c). Furthermore, students who 
seek mental health interventions represent a broad spectrum 
of severity and problem type. Although a small number of 
high-need youth may receive a disproportionate number of 
SMH sessions, research also suggests that most students 
present with subclinical problems and use fewer than four 
sessions per year (Lyon, Charlesworth-Attie, Vander Stoep, 
& McCauley, 2011; Lyon et al., in press; Walker, Kerns, 
Lyon, Bruns, & Cosgrove, 2010). Therefore, to maximize 
impact across this broad spectrum of needs, SMH interven- 
tions must be designed to be both effective and time efficient. 
SMH services must also be able to efficiently identify those 
students with more intensive needs—or those who do not 
respond to the intervention—and either modify the approach 
to service delivery or connect them with alternative, poten- 
tially more intensive, services. 

Driven by the previously described barriers to EBP 
implementation in schools and with explicit attention to 
contextual appropriateness, our research team is iteratively 
developing and testing a systematic, evidence-based inter- 
vention focused on assessment, problem-solving, and skill- 
building that can be implemented as a “Tier 2” approach 
within a larger framework of multi-tiered systems of sup- 
ports (MTSS; National Center on Response to Intervention, 
2010). The developing BRISC protocol (Lyon et al., 2014a), 
is informed by recent developments in common elements 
approaches to psychotherapy with youth (Chorpita, Dalei- 
den, & Weisz, 2005; Weisz et al., 2012), therefore includes 
identification of “top” problems, use of a problem-solving 
framework, and inclusion of routine progress monitoring 
using standardized and idiographic approaches (Borntrager 
& Lyon, 2015), and is intended to be applicable to the 
majority of student presenting problems commonly 
encountered in educational settings. 

With funding from the Institute of Education Sciences, 
we are engaged in an iterative development and testing 
process in public secondary schools. The overarching 
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project includes four primary phases: (1) expert and 
stakeholder input, (2) feasibility and acceptability testing in 
schools, (3) pilot testing with existing school-based prac- 
titioners, and (4) a small-scale randomized study. The 
current paper describes the second phase of this process 
(feasibility testing). For a full discussion of the theoretical 
and empirical foundations for the developing BRISC 
model and results of expert and stakeholder input, please 
see Lyon et al. (2014a). 

Studies conducted during Phase 1 of BRISC protocol 
development included an initial evaluation of BRISC’s 
appropriateness to the school context, based on the per- 
spectives of administrators, service providers, and national 
experts. As described by Lyon et al. (2014a), a number of 
crosscutting themes emerged to guide BRISC develop- 
ment: (a) be responsive to patterns of student help-seeking 
in schools (e.g., inconsistent attendance in therapy, use of 
four or fewer sessions); (b) develop methods or pathways 
for referrals based on student response to the intervention 
(including additional educational, social, or mental health 
services); (c) adopt a flexible and responsive stepped-care 
structure (e.g., a straightforward intervention approach 
that includes modules focusing on typical problem areas 
identified by students, facilitate client-centered decisions 
about the specific intervention focus); and (d) use educa- 
tional (e.g., attendance) and other forms of data to drive 
decision-making, including data on treatment targets that 
resonate with students (e.g., “stress”). Such input 
informed initial development of the BRISC intervention 
model and protocol. 


Implementation Outcomes: Feasibility, 
Acceptability, and Appropriateness 


In addition to supporting the development of an effective 
and contextually appropriate school-based intervention, a 
goal of Phase 2 was to evaluate key implementation pro- 
cesses and outcomes. Among the implementation outcomes 
articulated by Proctor et al. (2011), feasibility, acceptabil- 
ity, and appropriateness all represent early-stage variables 
with implications for the long-term success of an imple- 
mentation effort in schools. Feasibility, or the “extent to 
which a new treatment, or an innovation, can be success- 
fully used or carried out within a given agency or setting” 
(p. 69), reflects the intersection between the requirements 
for implementing a particular innovation and the con- 
straints present in the destination context. Acceptability is 
defined as “the perception among implementation stake- 
holders that a given treatment, service, practice, or inno- 
vation is agreeable, palatable, or satisfactory” (p. 67), 
which is best assessed through a stakeholder’s direct 
experience with a particular intervention or innovation. 
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Finally, appropriateness can be defined as “the perceived 
fit, relevance, or compatibility of the innovation or evi- 
dence-based practice for a given practice setting, provider, 
or consumer; and/or perceived fit of the innovation to 
address a particular issue or problem” (p. 69). 


Current Aims 


To inform continued development and revision of the 
BRISC protocol, the primary aim of the current paper is to 
use mixed qualitative and quantitative analyses to evaluate 
its feasibility, acceptability, and appropriateness in an 
authentic education setting, emphasizing student experi- 
ences. Data were collected in the second phase of the BRISC 
development project, during which university-based thera- 
pists from the research team delivered the intervention to 
high school students during the school day. We assessed the 
following research questions: (Research Question [RQ] 1) 
How well were clinicians able to deliver BRISC components 
(feasibility)? (RQ2) To what extent is the BRISC protocol 
acceptable to students (acceptability)? and (RQ3) To what 
extent did the BRISC protocol fit the presenting problems 
and preferences of referred students and promote improve- 
ment in students’ priority problem areas (appropriateness)? 


Method 
Participants 


Participants were 11 high school students who received the 
BRISC intervention in a single school during the 2012/2013 
academic year (out of 15 approached—see below). Ten 
participants were female (91 %). Six were Caucasian and not 
of Hispanic descent (55 %), two were African American, two 
were Hispanic/Latino, and one was African. Students rep- 
resented a mix of grade levels (one in ninth grade, two in 
tenth grade, three in 11th grade, and five in 12th grade). Four 
students (46 %) had failed one or more classes during the 
prior semester. Information on referral source was available 
for nine participants. Of those, five of the nine (56 %) were 
self-referred, two were referred by the nurse practitioner in 
the school-based health center, one by a teacher, and one by a 
school counselor. Enrollment was based on ongoing or 
“rolling” referrals throughout the spring semester of 2013. 


Procedures 
Research staff contacted students referred to BRISC to 
review details of the project, answer all questions, and 


invite them and a parent to participate in the intervention 
and evaluation. Of 15 approached, 12 agreed to participate; 
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one of these participants attended no BRISC sessions and is 
excluded from this report. The parents of the three students 
who had turned 18 were not approached to participate. 
Parents of two other participants under age 18 declined to 
participate, but allowed their student to participate. 
Research staff conducted baseline assessments in which 
they gathered demographic information and administered 
measures of mental health and adaptive functioning status. 
Data were also gathered on the number of sessions com- 
pleted, the length of the sessions, and the number of days 
from the first to last session. Therapists completed fidelity 
rating checklists after each of the four BRISC sessions 
documenting whether or not they implemented the inter- 
vention elements that were intended for each session. Eight 
weeks after the first BRISC intervention session, after the 
completion of BRISC participation, study team members 
met with students in person to conduct qualitative exit 
interviews and collect post-intervention measures (see 
Measures). To support open and honest reporting about the 
services received, all students were interviewed by research 
team members other than their BRISC therapist. All study 
procedures were approved by the institutional review board 
at the University of Washington. 


BRISC Intervention 


Although it is ultimately intended for use by existing 
school-based professionals, in the current study the BRISC 
intervention was delivered by one masters-level social 
worker and two doctoral-level psychologists, all research 
team members, to allow for an adequate “test of concept” 
prior to training embedded staff. Study therapists were 
involved in the development of the BRISC protocol and 
had substantial knowledge of its content and procedures. 
Therapists participated in biweekly group _ supervi- 
sion/consultation to review cases and troubleshoot diffi- 
culties encountered when delivering the protocol. 

To allow students to attend the BRISC sessions, the 
existing standard procedure for school-based health center 
appointments was used. That is, administrative staff sent 
passes to participating students’ classrooms, excusing them 
to attend their session. The BRISC protocol outlined four 
sessions (each lasting between 30 and 60 min), based on 
treatment elements shared across effective interventions for 
internalizing and externalizing problems in adolescents 
(see Table 1). Session one included administration of and 
feedback on brief standardized assessment measures (Pa- 
tient Health Questionnaire [PHQ-9] and Generalized 
Anxiety Disorder scale [GAD-7]—see Measures), psy- 
choeducation about the BRISC intervention, a brief func- 
tional assessment (targeting school, family, peers), problem 
identification, stress rating, and introduction to informal 
symptom or behavioral monitoring. Session two included 
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monitoring review; psychoeducation on anxiety, depres- 
sion, and stress; and introduction to problem-solving, 
focusing on the identified top problem. 

Session three was individualized based on identified 
problem and barriers to solution implementation. Therapists 
could choose from four individual modules that included: 
“Dealing with a hard situation I can’t change” (cognitive 
restructuring), “I just don’t feel like it” (motivation 
enhancement), “Getting along with other people” (commu- 
nication skills), and “Handling hard feelings” (stress and 
mood management). Session four involved collaborative 
review of progress and identification of next steps: (a) Return 
for SMH services if the student determines it is indicated (1.e., 
“come back if you need it”). In these cases, the presenting 
concern/problem is resolved and both therapist and student 
agreed no more care was indicated. However, the student is 
reminded that they can come back to see the counselor again in 
the future. (b) Supportive monitoring—regular brief check-ins 
with the therapist or, ideally, another school-based helper (i.e., 
teacher, coach, school counselor). In these cases, the pre- 
senting problem was resolved, but the therapist and student 
agreed that checking in with a supportive person in the school 
would help them stay on track and continue to progress. 
(c) Continued BRISC or other treatment from the SMH pro- 
vider (e.g., therapist continues to meet regularly with the 
student). (d) Referral to more intensive services (often exter- 
nal to the school). In addition, BRISC involved the use of 
engagement strategies, informed by the growing literature on 
motivational enhancement (Miller & Rollnick, 2012), to 
facilitate student involvement in the intervention process and 
reinforce help-seeking behavior. 


Measures: Intervention Delivery and Acceptability 


BRISC session data Therapists documented number of 
sessions scheduled, attendance (vs. no show/cancelation), 
and session length. 

Therapist adherence checklist A preliminary adherence 
checklist for the developing BRISC intervention was 
completed by study therapists following each BRISC ses- 
sion in which they identified which core BRISC elements 
they had delivered (e.g., administered standardized assess- 
ments, identified top problems, assigned practice exercises). 
Each of the BRISC sessions had a different number of 
recommended intervention elements, ranging from 10 
(Session 3) to 14 (Session 1). Therapists also used adher- 
ence checklist to record the student’s presenting problem(s). 

Multidimensional Adolescent Satisfaction Scale (MASS; 
Garland, Saltzman, & Aarons, 2000), a 21-item measure of 
client satisfaction with the services they receive from their 
mental health provider, was administered at the eight-week 
(post-BRISC) exit interview to assess student acceptability. 
The MASS has four factors—counselor qualities, meeting 
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Table 1 Content of the BRISC 


protaael Session # 


Session | 


Session 2 


Session 3 


Session 4 


needs, effectiveness, counselor conflict—and has previ- 
ously demonstrated strong psychometric properties (Gar- 
land et al., 2000). 


Exit Interview 


Eight weeks after their first BRISC session, participating 
students completed a semi-structured qualitative exit 
interview in addition to answering standardized question- 
naires. Open-ended questions addressed reasons for seeking 
services, perceived BRISC utility, and feedback on specific 
aspects of BRISC (e.g., assessment tools, homework/prac- 
tice exercises). During the interview, students were also 
asked to rate their agreement with several statements about 
whether they received and/or benefitted from different 
components of the intervention (e.g., practice/homework 
activities) using a Likert-style scale ranging from 1 to 10. 
Students also retrospectively rated their level of motivation 
to engage in BRISC initially and then to attend the second 
session, third session, and final fourth sessions. 


Measures: Student Symptoms and Outcomes 


Behavior Assessment System for Children-2 (BASC-2; Rey- 
nolds & Kamphaus, 2004). The BASC-2 is a well-established 


Session content 


Administration and feedback of standardized assessment (SA) 
Introduction to BRISC 
Brief functional assessment (targeting school, family, peers) 
Problem identification 
Complete “stress” rating 
Introduction to informal monitoring 
Review monitoring 
Psychoeducation stress/mood 
Identification of top problem 
Introduction to problem-solving 
Select solution and set up behavioral experiment for practice exercise 
Review practice exercise 
Select and implement module 
“Dealing with hard situations I can’t change” (cognitive restructuring) 
“I just don’t feel like it” (motivation enhancement) 
“Getting along with other people” (communication skills) 
“Handling hard feelings” (stress and mood management) 
Assign practice exercise based on selected module 
Review practice exercise 
Administration and feedback of SA 
Review progress 
Identify what is still needed 


Discuss next steps 


measure designed to assess behaviors and emotions among 
children and adolescents, including anger control, bullying, 
emotional self-control, negative emotionality, and resiliency. 
Internal consistencies of the BASC-2 subscales range from 
a = .76 to « = .95 for adolescents. Test—retest reliability is 
.81 (Reynolds & Kamphaus, 2004). The BASC-2 was 
administered only at baseline and used to describe participants 
as part of our effort to assess appropriateness of BRISC to the 
needs of students. 

Revised Ways of Coping Checklist (RWCC; Vitaliano, 
Russo, Carr, Maiuro, & Becker, 1985). The RWCC is a self- 
report revision of the Ways of Coping Checklist developed 
by Folkman & Lazarus (1988) that assesses different 
methods of coping with stress. The RWCC includes eight 
subscales, four of which represent negative ways of coping 
(confrontive coping, distancing, self-controlling, escape 
avoidance) and four representing positive ways of coping 
(seeking social support, accepting responsibility, planful 
problem-solving, and positive reappraisal). The Revised 
Ways of Coping Checklist (RWCC) is scored on a relative 
scale such that each approach to coping is considered as a 
proportion of an overall “100 %” coping in which the 
respondent engages. The RWCC has been used successfully 
with adolescents in high school with good internal consis- 
tency across subscales (« = .69-.96) (Wagner, Myers, & 
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Mcininch, 1999). Change over time is measured as the 
change in proportion of total coping for each different 
approach to coping. The RWCC was administered at base- 
line and eight-week follow-up. 

Columbia Impairment Scale (CIS; Bird, Andrews, & 
Schwab-Stone, 1996). The 13-item CIS was administered 
to adolescents to measure overall level of adaptive func- 
tioning. The scale was used in the NIMH Methods for the 
Epidemiology of Child and Adolescent Mental Disorders 
(MECA) study and showed a reliability of « = .83 for 
student report and good validity (Bird, Shaffer, Fisher, & 
Gould, 1993). It is also correlated with other indicators of 
psychological dysfunction and with the clinician-rated 
Children’s Global Assessment Scale (Bird, Canino, Rubio- 
Stipec, & Ribera 1987). The CIS was administered at 
baseline and eight-week follow-up. 

Patient Health Questionnaire (PHQ-9). The PHQ-9 is a 
widely used brief scale that queries about the presence of 
nine criteria for major depressive disorder and has high 
sensitivity (88 %) and specificity (88 %) (Kroenke, Spit- 
zer, & Williams, 2001; Léwe, Kroenke, Herzog, & Grafe, 
2004). Research supports its use with adolescents, includ- 
ing good sensitivity (90 %) and specificity (79 %) in this 
population (Richardson et al., 2010). The PHQ-9 was 
administered by BRISC therapists to students in BRISC 
Session | and again in BRISC Session 4. 

Generalized Anxiety Disorder Scale (GAD-7; Spitzer, 
Kroenke, Williams, & Lowe, 2006). The GAD-7 is a brief 
scale that queries about symptoms for generalized anxiety 
disorder. Normative data for the GAD-7 included a large 
sample of adolescents aged 14-25 years (n = 634) and 
concluded that the measure demonstrated age invariance. 
Internal consistency was found to be « = .89 (Lowe et al., 
2008). Similar to the PHQ-9, the GAD-7 was administered 
by BRISC therapists to students in BRISC Session 1 and 
again in BRISC Session 4. 


Analytic Approach 
Quantitative Analyses 


Descriptive statistics were calculated for adherence as 
recorded by therapists, student common presenting problems 
(as recorded by therapists on the adherence checklist), and 
intervention delivery. For quantitative measures that were 
administered repeatedly (e.g., student symptoms, function- 
ing, coping), simple ¢ test or ANOVA comparisons were 
made between pre-—post scores. Effect sizes were calculated 
as within-subject Cohen’s d (Cohen, 1988) using pooled 
standard deviations and Morris & DeShon’s (2002) equa- 
tion 8 for repeated measures. To provide a general bench- 
mark for comparison, client satisfaction scores from the 
MASS were compared to scores from a different sample of 
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adolescents who had received outpatient community-based 
mental health services as reported in a psychometric study of 
the MASS (Garland et al., 2000), with effect sizes calculated 
as between-subjects Cohen’s d (Cohen, 1988). Effect size 
threshold interpretations follow Cohen’s (1988) recom- 
mendations with small effects between .20 and .50, medium 
effects between .50 and .80, large effects between .80 and 
1.30, and very large effects above 1.30, while acknowl- 
edging the inherent risks involved in applying these simple 
categorizations without additional context (c.f. Glass, 
McGaw, & Smith, 1981). 


Qualitative Analyses 


Qualitative interviews were audio recorded, transcribed, 
and then coded using conventional and directed content 
analysis (Hsieh & Shannon, 2005) and qualitative coding 
software (Atlas.ti; Friese, 2012), which allows research 
team members to highlight text segments and assign codes. 
A team of four coders reviewed student responses to each 
question from the same subset of transcripts, identified 
potential codes, and then met to produce an initial code- 
book. The resulting codebook was trialed independently by 
coders across additional transcripts and then revised. This 
process continued over several iterations until a stable set 
of codes was reached. Coding used a consensus process in 
which each transcript was re-coded independently by all 
four raters who then met to arrive at consensus judgments 
through open dialog (DeSantis & Ugarriza, 2000; Hill 
et al., 2005). Portions of the second and third research 
questions were also evaluated using a mixed-methods 
approach for the purposes of data elaboration, explanation, 
and triangulation (Palinkas et al., 2011). 


Results 


In the following sections, we address each research ques- 
tion using a combination of quantitative and qualitative 
data. Tables 2 and 3 present key quantitative variables, and 
Table 4 presents the qualitative code hierarchy. 


Research Question 1: How Well were Clinicians able 
to Deliver BRISC Components (feasibility)? 


The extent to which study clinicians were able to deliver 
BRISC was evaluated via data on session length, total 
duration of BRISC service delivery, and the therapist 
adherence checklist (see Table 2). It took an average of 
27 days to complete all four sessions (approximately 1 
session/week), and despite a wide range (21-60 min), most 
sessions were approximately 50 min. Data from the 
adherence checklist indicated that clinicians generally 
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Table 2 BRISC feasibility and acceptability 
Median (n = 11) Range 

Feasibility 
Days between first and last session, for those completing four sessions 27 21-42 
Session length (min) 50 21-60 
Fidelity (percentage of items) 
Session | fidelity 85.7 % (12/14) 78.6-100 % 
Session 2 fidelity 90.9 % (10/11) 54.5-100 % 
Session 3 fidelity 80 % (8/10) 10-90 % 
Session 4 fidelity 72.7 % (8/11) 54.5-90.9 % 

BRISC Mean Range Comparison mean =p Cohen’s d 
Acceptability 
Multidimensional adolescent satisfaction scale total mean (1-4) 3.44 (.25) 3.19-4.00 3.12 (.89) .002 561 
Counselor qualities 3.67 (.22) 3.33-4.00 3.21 (.87) <.001 844 
Meeting needs 3.30 (.52) 2.27-4.00 2.84 (.99) .016 .609 
Effectiveness 2.66 (.67) 1.75-4.00 2.86 (.97) 346 = =—.280 
Counselor conflict (reverse scored, higher is less conflict) 3.89 (.17) 3.50-4.00 3.47 (.76) <.001 .903 
Motivation (1-10)? Mean Range Cohen’s d for change from prior session 
Session | 6.27 (2.41) 1-10 N/A 
Session 2 8.27 (1.19) 6-10 928 
Session 3° 8.60 (1.08) 7-10 372 
Session 4° 8.89 (1.17) 7-10 331 
Homework/practice activity helpfulness 7.50 (1.65) 6-10 


“ Comparison data for single-sample t-tests obtained from Garland, Saltzman, & Aarons, 2000 


> Repeated-measures ANOVA Fu3) = 5.86, p = .03, corrected for lack of sphericity 


“n= 10 


4n=9 


reported being able to deliver most session components, 
with median delivery of components ranging from 73 to 
91 % by session. 


Research Question 2: To What Extent is the BRISC 
Protocol Acceptable to Students? 


Quantitative Data 


Acceptability was first evaluated by examining session 
attendance and mean MASS total and subscale scores. Of 
the 11 participants who received at least one session, nine 
(82 %) attended all four sessions, one dropped out after 
two sessions, and another reported clinical improvement 
and no further need for treatment, after only three sessions. 
Single-sample t-tests were used to compare BRISC MASS 
results to results from another sample of adolescents 
(n = 180) receiving “usual care” services in outpatient 
settings (Garland et al., 2000). Table 2 shows that BRISC 
participants were significantly more satisfied overall as 
assessed by the MASS total score, with a medium effect 


(d = .561). BRISC participants also reported higher satis- 
faction on three of four subscales with medium to strong 
effects, including higher ratings of counselor qualities 
(d = .844), higher ratings of counselor ability to meet their 
needs (d = .609), and lower ratings of counselor conflict 
(d = .903). There were no significant differences on the 
MASS effectiveness subscale. 

A repeated-measures ANOVA found exit interview 
ratings of motivation to attend sessions significantly 
increased over all sessions (F(7.3) = 5.86, p = .03, cor- 
rected for lack of sphericity). Within-subject tests indicated 
that the change in motivation was largely attributable to the 
first and second sessions; for the nine students who com- 
pleted BRISC, mean ratings of self-reported motivation to 
attend BRISC sessions (out of a possible 10 points) were 
6.3 for the initial session and then increased to 8.3 for the 
second session (post hoc within-subject F(z,s) = 6, 
p = .04, d = .928), 8.7 for the third session (F(;,s) = 1.3, 
p = .28, d= .372), and 8.9 for the fourth session 
(Fas) = 1, p = 347, d= .331). The helpfulness of 
homework/practice was rated a mean of 7.5. 
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Table 3 BRISC appropriateness (symptoms and outcomes) 
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Pre-intervention mean Post-intervention mean F df D Cohen’s d 
proportion (SD) proportion (SD) 
Revised ways of coping omnibus test 33 74 3.3 
Confrontive 105 (.049) .094 (.041) 0.98 1,10 345 —.306 
Distancing 132 (.051) 106 (.035) 2.63 1,10 136 —.504 
Self-controlling .167 (.025) 152 (.059) 0.93 1,10 358 —.354 
Seeking social support -110 (.036) .146 (.070) 4.01 1,10 073 .697 
Accepting responsibility 162 (.049) .137 (.066) 1.74 1,10 217 —.408 
Escape avoidance 118 (.048) .095 (.063) 1.79 1,10 211 —.415 
Planful problem-solving 113 (.048) 156 (.083) 6.10 1,10 .033 895 
Positive reappraisal .092 (.033) 114 (.052) 1.79 1,10 .210 421 
Pre-intervention mean (SD) Post-intervention mean (SD) t df D 
PHQ-9 (N = 10) 7.3 (7.4) 6.4 (6.59) 73 484 = 235 
GAD-7 (N = 10) 4.5 (5.4) 5.1 (4.7) —47 653 149 
CIS (VN = 11) 17.4 (8.8) 12.6 (8.6) 1.49 10 168 —.449 


Table 4 Qualitative codes from BRISC student interviews 


Code Brief description # Mentioning (of 11) 

Engage High or low motivation to participate in BRISC 11 

Barriers Barriers to BRISC participation 3 

Improve Areas in which BRISC could be improved 9 

Problem/ Personal student problems for and outcomes of BRISC services Mentioned as Mentioned as 

outcome problem outcome 

Relationship Interpersonal relationships 2 2 

Mood/anxiety | Mood or anxiety issues 4 3 

Sleep Sleep/sleep hygiene 3 2 

Academic Educational issues/outcomes (academic and school) 4 2 

Anger Emphasis on anger problems 3 2 

Service Connection to other services, typically addressed through linkages/referrals/case 0 3 
management 

Stress References to experiencing “stress” that is not explicitly tied to another problem 3 1 
category 

Eating References to problems with eating/diet 2 0 

Skill Benefiting from BRISC by developing/learning new skills 0 4 

Post-BRISC Comments about/relevant to post-BRISC pathways 7 


Qualitative Data 


Student qualitative interviews yielded codes that provided 
additional insight into the acceptability of BRISC (see 
Table 4). Specifically, codes captured student comments that 
reflected reasons for high or low engagement in BRISC, 
identified barriers to participation, and cataloged specific 
ways in which students suggested that the BRISC inter- 
vention could be improved. Of these codes, engagement 
comments were the most common and generally reflected 
high motivation to participate, consistent with quantitative 
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data for BRISC sessions 2-4. Some students indicated that 
their engagement was driven by a basic desire to get help for 
their problems (“I was really depressed...just worried for 
my well-being, so I went to see someone for it”). Other 
comments focused on the motivational value of therapist 
characteristics (e.g., “she was really kind,” “a good lis- 
tener,” “I felt like she really enjoyed her job and wanted to 
make a difference”). Consistent with lower motivation rat- 
ings for the first session than for subsequent sessions, some 
students reported beginning BRISC with hesitation (“I kind 
of had my reservations about completely...letting it all out 


School Mental Health (2015) 7:273-286 


there to talk to somebody”). However, motivation increased 
as they attended additional sessions (“...it was a lot more 
productive than I thought it would be at the beginning and I 
sort of liked that...and I liked coming here”). 

Specific practical and social barriers to participation 
were only mentioned by three participants and included the 
drawback of missing class, social stigma, and family 
pressures (e.g., “I didn’t want my dad to know,” “My 
parent didn’t want me to”). These barriers appeared to 
relate to mental health intervention in general, rather than 
the BRISC intervention, specifically. 

Students also made a number of comments about ways 
to improve BRISC, based on their experience during the 
pilot, and presumably increase its acceptability. Although 
BRISC was designed to be a targeted initial assessment and 
engagement approach within a tiered service model, some 
comments reflected student interest in an intervention of 
greater breadth, depth, or length (e.g., “I just think one 
more session would have helped because we would’ve 
been able to see how what we did worked over the longer 
term,” “with extra sessions, you get more closure”). In 
addition to more sessions, this included an expressed desire 
that therapists focus on more than one target issue, and 
“more options or alternatives for the activities.” Never- 
theless, other students commented that the duration of 
services was likely sufficient (“I think [more sessions] 
would have continued to be helpful, but it’s not a necessity 
for me to keep seeing her”). Finally, the largest number of 
improve codes was concrete, emphasizing ways to make 
the handouts clearer or more engaging (“I would change 
them to make them very diagramy-type, not just words on a 
page”). However, they did not seem to interfere with 
adequate service provision (e.g., “the handouts, a lot of 
them were just general, but we would like, flip it over, and 
write out specifically what’s going on’). 


Research Question 3: To What Extent did 

the BRISC Protocol fit the Presenting Problems 
and Preferences of Referred Students 
(Appropriateness)? 


Presenting Problems 


Participant BASC self-ratings at baseline indicated that 
seven of the 11 participants (63.6 %) had at least one 
clinically significant elevation in a problem area. Three of 
the seven had one elevation, two had two elevations, one 
had three, and one had nine clinically significant elevations 
in problems areas on the BASC. Internalizing problems, 
such as depression, anxiety, sense of inadequacy, and 
somatization, were the most problems endorsed at a clini- 
cally significant or at-risk level by BRISC participants. 
Therapist adherence forms provided additional information 
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about the identified primary problem areas for intervention. 
Data from adherence forms indicated that BRISC treatment 
focused on one single problem area for nine participants 
and on two primary problem areas for two students. 
Problem areas included academic difficulties (five partici- 
pants), depression (three participants), peer problems (three 
participants), anxiety (one participant), truancy (one par- 
ticipant), and sexual trauma (one participant). Qualitative 
problem codes were also used to catalog student descrip- 
tions of the reasons why they sought treatment and both 
confirmed and expanded on the clinician-identified prob- 
lems. Similar to the above, students were most likely to 
describe academic difficulties (four participants), mood/ 
anxiety problems (four participants), and interpersonal re- 
lationship issues (two participants). In addition, students 
referenced a general experience of “stress” (three partici- 
pants), sleep difficulties (three participants), anger (three 
participants), eating problems (two participants), and time 
management (one participant). 


Improvement in Coping, Symptoms, and Functioning 


Given the small sample size, lack of a control or contrast 
group, and brevity of the intervention, only a very pre- 
liminary assessment of change over time is appropriate. We 
did, however, explore change over time on key outcome 
measures (see Table 3). RWCC scores from pre- and post- 
intervention were analyzed using a repeated-measures 
ANOVA. An omnibus F test for within-subject change was 
nonsignificant (F(74 = 3.3, p = .133), but exploratory 
post hoc tests did provide positive support for a significant 
increase in the relative use of planful problem-solving, 
which had a large effect size (d = .895), and a borderline 
significant increase in seeking social support, which had a 
medium effect size (d = .697). Distancing also had a 
medium effect size decrease in relative use (d = —.504). 
Paired t-tests yielded no significant pre—post changes in the 
PHQ-9, GAD-7, or the CIS, although the CIS approached a 
medium effect size (d = .449) and the PHQ had a small 
effect size (d = .235) in the expected, positive direction. 
Student descriptions of the gains they experienced over 
the course of BRISC services were also evaluated quali- 
tatively. Outcome coding began with the identified set of 
problem codes described above, but was also expanded— 
based on student responses—to include two additional 
outcomes: connection to other services (i.e., via referrals, 
case management, etc.) and the development of new skills 
(e.g., “a new method of thinking,” “I established this new 
technique for myself to shut up, chill, and listen”). With 
the exception of the eating problem code, all of the prob- 
lem codes described above were also identified as outcome 
codes for at least one student (see Table 4). Mood/anxiety 
problems (four participants), skills (four participants), and 
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service linkages (three participants) were the most com- 
monly mentioned outcomes. A within-subject comparison 
of the qualitative outcome codes with the problem codes 
indicated that, across problems, approximately half of the 
students who sought services for a particular problem 
indicated improvement in that specific problem. However, 
students rarely reported gains in areas that were not iden- 
tified as problems at the outset of BRISC (problem codes), 
with the exception of skills and services. 

Finally, to shed further light on the appropriateness of 
the BRISC protocol and inform the development of a 
number of pathways for service connection or monitoring 
following the conclusion of BRISC, we evaluated the 
additional service needs identified for participating stu- 
dents. Session records indicated that, of the nine students 
who completed four BRISC sessions, two were referred to 
an existing school-based mental health counselor for fur- 
ther support, two were referred to external community care 
providers for more intensive services, two were invited to 
continue checking in with school-based counselors on an 
“as needed” basis, and three were not referred for further 
services. Student descriptions of their plans for services 
post-BRISC were similarly varied, including some stating 
that they “didn’t feel like there was more I wanted to work 
on” unless “another problem were to arise,” and others 
indicating that there was “more in terms of my grades and 
school work [on which I would like to keep working] also, 
just trying to get my mood up”). 


Discussion 


The current study represented the second phase of an 
ongoing iterative BRISC development process. Results 
provide preliminary evidence that BRISC was feasible in 
the school setting, given that clinicians reported delivering 
73-91 % of session components and the vast majority of 
students were able to complete the prescribed four sessions. 
Students’ ratings of satisfaction and motivation, along with 
positive comments from interviews, also indicated it was 
acceptable. Finally, through the use of BRISC assessment 
tools and structure, clinicians were able to accurately 
determine a student-relevant problem area for focus, 
address many of the presenting problems exhibited by 
youth, and determine a diverse set of seemingly appropriate 
pathways for follow-up care for students after these four 
sessions, providing additional evidence of BRISC’s 
appropriateness to the school setting. Results also pointed 
to a number of opportunities to improve the developing 
BRISC protocol. Below, we discuss key findings as they 
relate to the continued development of BRISC and to SMH 
in general. 
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Improving Student Satisfaction and Engagement 


Student participants seem to have been satisfied with the 
BRISC intervention, with engagement scores higher than 
those from a separate, large adolescent sample (Garland 
et al., 2000) and a moderate effect size for total satisfac- 
tion. However, without a local comparison group, it is 
unclear whether satisfaction was specifically related to the 
BRISC intervention. These quantitative findings are sup- 
plemented by our qualitative work, suggesting that students 
found BRISC sessions productive and helpful. Both 
quantitative and qualitative analyses indicate that motiva- 
tion to participate in BRISC seemed to increase over the 
four sessions (with the largest effects occurring between 
Session | and Session 2), suggesting that BRISC activities 
were relevant to students across the short duration of 
treatment and that some of this relevance may have been 
quickly apparent to participants. 

Nevertheless, student feedback and clinician observations 
suggest the first and second sessions of BRISC may still need 
to be refined to provide more opportunities for engagement, 
and for students to immediately see the potential utility of 
treatment. In BRISC development, we used motivational 
interviewing strategies (Miller & Rollnick, 2012) to build 
client motivation to engage in treatment, beginning in the first 
session (e.g., differentiating between confidence [“it won’t 
work” ] and importance [“I forgot” ] cues and asking corre- 
sponding “ruler” [0-10 questions] to elicit change talk). 
Options for additional strategies to improve engagement and 
motivation may be found in recent structured reviews. For 
example, Lindsey et al. (2014) found five major families of 
engagement practice elements. Two of these strategies are 
inherent to BRISC as a school-based intervention (accessi- 
bility promotion and appointment reminders), and others are 
already built in, including routine homework assignment and 
the use of a brief but structured assessment approach. Based on 
this feasibility trial, we have revised our protocol to enhance 
the final type of engagement element identified by Lindsay 
et al. (2014), psychoeducation about the treatment, in which 
adolescents are given a clear game plan about working toge- 
ther for 34 sessions to address a problem they have identified, 
followed by a check into determine next steps. 

In addition to continuing to refine existing BRISC 
engagement strategies, the current pilot illuminated the 
potential benefit of streamlining and simplifying BRISC. 
Given that study clinicians were able to deliver the inter- 
vention within the recommended time (four 30-60 min 
sessions) and that nine of 11 students attended all four 
sessions, BRISC seem feasible. However, data from the 
adherence checklist indicated that study clinicians were not 
always able to deliver all of the session components (fidelity 
ranged from 73 to 91 %). These data suggest BRISC may 
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still require further simplification, especially given that the 
clinicians in this pilot test were university-affiliated thera- 
pists who were well oriented to the BRISC purpose and 
approach and likely had more “bandwidth” to implement 
the intervention. Such a finding is typical of initial pilot tests 
and aligns with calls by dissemination and implementation 
researchers to simplify interventions to increase likelihood 
of delivery in public health settings (e.g., Rotheram-Borus, 
Swendeman, & Chorpita, 2012). 


Meeting Student Needs 


One of the main goals for BRISC development was to 
design a “stand-alone” intervention that—in keeping with 
the contextual constraints of the education sector (Lyon 
et al., 2014a)—-could be delivered in four or fewer sessions, 
with options for post-BRISC triage/planning that places 
BRISC as an entry point into a tiered service model. To 
achieve meaningful change within such a brief timeframe, 
problem-solving was incorporated as a cornerstone of the 
BRISC intervention. BRISC clinicians are trained and coa- 
ched to work with students to carefully articulate what they 
are hoping to change as a result of their participation. For- 
tunately, toward this end, results of the study found con- 
sistency between clinician and student reports of presenting 
problems, suggesting that clinicians were able to accurately 
identify student concerns, or, at the very least, the BRISC 
process frequently yielded a shared understanding of 
mutually identified problems between provider and service 
recipient. This is particularly important in light of research, 
indicating that a large portion in variance in therapy out- 
comes is accounted for by clearly defined goals that are 
shared by therapist and client, combined with progress 
monitoring toward those goals (Carlier et al., 2012; Lambert 
et al., 2003). Also important for feasibility and acceptability, 
data suggest that BRISC was often used to target common 
school-interfering conditions, with academic difficulties 
among the most commonly identified reasons for referral 
across information sources. Continued emphasis on aca- 
demically relevant problems is likely to enhance the con- 
textual appropriateness of the final BRISC protocol (Lyon, 
Borntrager, Nakamura, & Higa-McMillan, 2013). 

In addition to identifying problems, four BRISC par- 
ticipants specifically mentioned learning new skills as a 
positive outcome. This is important considering the explicit 
emphasis of BRISC on skill development within the lim- 
ited service delivery timeframe. Across problems, 
approximately half of the students who sought services for 
a particular problem indicated that the problem improved. 
Qualitative findings provided general support for skill 
development as an outcome for BRISC and findings from 
quantitative outcomes suggest that student participants had 
significant improvement on one specific measure of coping 
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skills, planful problem-solving. Indeed, while other sub- 
scales from our coping measure demonstrated small to 
medium effect sizes, planful problem-solving demonstrated 
a large effect size (d = .895). Given the substantial focus 
of BRISC on problem-solving, this improvement is 
promising. Although no other outcomes showed significant 
change from pre- to post-BRISC, all quantitative measures 
(except the GAD-7, which increased on average by one 
point) showed trends toward improvement (with small 
effect sizes). Results from our Phase 3 and 4 BRISC studies 
will help to confirm these trends with larger samples and 
representative providers. 


Next Steps: Enhancing BRISC’s Potential 
for Positive Impact 


In addition to our iterative efforts to improve existing BRISC 
components, the pilot test has also led the research team to 
appreciate BRISC’s potential viability as an engagement 
strategy unto itself. Although BRISC was created to provide 
an option in the tiered service system that is often missing for 
many students—a brief, stand-alone intervention tailored to 
the school context—BRISC may serve more as an entry point 
into mental health services. Session records and qualitative 
data on students’ expressed needs indicated that six of the 
nine students who completed four BRISC sessions may 
benefit from additional mental health service supports. 
Moreover, in exit interviews, three students listed service 
linkages as a positive outcome of BRISC. 

Although this iteration of BRISC was not intended to be 
an engagement and triage strategy, we are increasingly 
considering that this may be one of its primary contributions. 
As we Strive to bring structure, efficiency, and research- 
based practice to SMH, it may be that, for some students, 
BRISC’s optimal role is to help identify their highest pri- 
ority needs, understand the nature and potential utility of 
psychosocial intervention, intervene initially, and, when 
necessary, connect them to subsequent treatment options 
that may be more intensive. As described above, engage- 
ment, problem identification, and initial problem-solving 
seem to be potential strengths of BRISC, and there is very 
preliminary evidence that BRISC may promote positive 
coping skills and positive improvements in functioning. At 
the same time, there also seems to be shared agreement 
between therapists and students that planning for next steps 
and finding the most appropriate post-BRISC pathway is an 
equally important part of the BRISC intervention. 

Future iterations of BRISC may highlight this compo- 
nent by introducing discussion of subsequent supports 
earlier and more explicitly, and when appropriate (e.g., in 
the case of elevated problem severity, but low likelihood of 
engagement in subsequent care), applying the motivational 
enhancement strategies currently embedded in the BRISC 
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intervention to support continued service engagement. The 
BRISC motivational enhancement component shares many 
similarities with existing approaches to improving client 
motivation to participate in family-focused mental health 
services (e.g., Participation Enhancement Intervention; 
Nock & Kazdin, 2005; Family Check Up; Dishion, Nelson, 
& Kavanagh, 2003) including its roots in similar founda- 
tional motivational enhancement therapies and its use of 
empathic, person-centered communication _ strategies. 
However, unlike many of the other approaches, BRISC is 
(a) designed to be a stand-alone intervention strategy for 
low-severity students, (b) intended to be an appealing “first 
encounter” with mental health services for students who 
may require more intensive services at a later point in time, 
and (c) designed specifically for use in the high school 
context. 


Limitations 


Limitations of the current study include that the findings 
were derived from a small sample of students who had 
received a preliminary version of the BRISC protocol in a 
single school. In addition, all findings were derived from 
student report (e.g., about coping strategies), which may be 
compared to the perceptions of others (e.g., parents, teach- 
ers) in subsequent trials. Although effect sizes were calcu- 
lated and simple tests of significance were conducted for 
some quantitative variables, interpretation of these analyses 
must be undertaken with caution, as should any general- 
ization of the findings beyond the current intervention 
development project. Furthermore, therapists were members 
of the BRISC development team and, as noted earlier, are 
not representative of typical SMH service providers. An 
additional limitation is lack of formal adherence coding. 
Development of BRISC fidelity tools was part of the pilot 
project described here. Therefore, although therapists com- 
pleted a trial adherence tracking form, sessions were not 
recorded or independently rated for adherence to the pro- 
tocol. BRISC studies are now underway with school-based 
clinicians, allowing for a primary focus on practitioner 
perspectives and needs. In this work, sessions are recorded 
and are being coded for adherence. Although we recognize 
that adherence to the BRISC protocol may be more varied 
when delivered by therapists who are not part of the study 
team, our goal is to develop an intervention protocol that can 
be easily used by a variety of school-based professionals and 
therefore includes a limited number of simple to track ele- 
ments that can be readily delivered to a youth across a 
spectrum of problems and tailored to suit their individual 
needs. Finally, although the current project incorporated a 
range of qualitative and quantitative data sources to address 
acceptability, feasibility, and appropriateness, standardized 
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tools to measure these constructs would have helped to 
better situate the BRISC protocol in the context of other 
psychosocial interventions. The lack of well-established 
implementation measures has been well documented (Mar- 
tinez et al., 2014). Fortunately, emerging work has begun to 
categorize existing measures and identify priorities for 
improvements (Lewis et al., 2015). 


Conclusion 


The current paper represents the second phase (feasibility 
testing) in an iterative intervention development process. 
By design, this phase was intended to yield information to 
drive further protocol refinement and testing in subsequent 
phases in more generalizable contexts. Findings were 
generally favorable surrounding the acceptability, feasi- 
bility, and appropriateness of the protocol, but also iden- 
tified multiple areas for potential improvement. For these 
reasons, the BRISC protocol and its accompanying mate- 
rials (e.g., training curriculum, adherence tool) will con- 
tinue to evolve. It is our hope that continued testing and 
refinement will ultimately lead to a service strategy that 
can successfully fill a critical need for schools, and con- 
tribute to the capacity of the SMH field to bolster the well- 
being of children and youth. 
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